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Abstract 

This paper provides two general classes of multiple decision functions where each member 
of the first class strongly controls the family- wise error rate (FWER), while each member 
of the second class strongly controls the false discovery rate (FDR). These classes offer the 
possibility that an optimal multiple decision function with respect to a pre-specified criterion, 
such as the missed discovery rate (MDR), could be found within these classes. Such multiple 
decision functions can be utilized in multiple testing, specifically, but not limited to, the 
analysis of high-dimensional microarray data sets. 
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1 Introduction 

Consider the situation which arises in the analysis of high- dimensional data, epitomized 
by a microarray data, where M pairs of null and alternative hypotheses, {Hmo,Hmi),rn = 
1,2,..., M, are simultaneously tested; see, for instance, O |6] for concrete examples of such 
situations. Two commonly-used Type I error rates for this multiple testing problem are 
the family- wise error rate (FWER), which is the probability of at least one false discovery, 
where discovery means rejecting (accepting) a null (an alternative) hypothesis, and the false 
discovery rate (FDR), which is the expectation of the ratio of the number of false discoveries 
over the number of discoveries. The usual testing paradigm employed in these situations 
is to decide on the collection of statistical tests for the M pairs of hypotheses, e.g., a t- 
test for each pair, obtain the p- value for each test, and then use the resulting M p- values 
in the FWER-controUing sequential Sidak procedure, provided an independence condition 
is satisfied, or the FDR-controlling procedure in [I]. In this conventional approach, there 
appears to be no leeway in the choice of the multiple testing procedure the moment the 
individual test procedures have been chosen. 

However, we pose the following question. If we are given the M test procedures for each 
of the M pairs of hypotheses, could we obtain classes of multiple testing procedures whose 
elements either control the FWER or the FDR? If the answer to this question is in the 
affirmative, then we may be able to find a multiple testing procedure within these classes 
which is optimal with respect to some chosen Type II error rate. And, we may then be able 
to choose the starting collection of test functions that will provide the best multiple testing 
procedure. 

This paper is in this spirit. We will demonstrate that, under certain conditions, when 
given a collection of test functions for the M pairs of hypotheses, that we can generate 
classes of multiple testing procedures controlling the FWER or the FDR. The results have 
important implications in the search for optimal multiple testing procedures that control 
either of these Type I error rates as we will see later. We shall investigate these issues in a 
general, but not surprisingly, more abstract framework. The main results in this paper were 
motivated by those in [TJl US] which did not deal with classes of multiple testing procedures, 
but instead focussed in developing improved FWER and FDR-controlling procedures from 
the Neyman- Pearson most powerful tests for each of the M pairs of hypotheses. 



2 Mathematical Setting 

Let {X,J',V) be a statistical model, so {X,J') is a measurable space and P is a collection 
of probability measures on {X,J^). Though not needed in the abstract development, for 
concreteness we may adopt the usual interpretation that X is the space of possible realizations 
of an observable random entity X from an experiment or a study. In decision problems with 
action space 21 = {0, 1}, such as in hypothesis testing, a nonrandomized decision function is 
a 6 : {X,J^) — )■ (21, (t(21)). In the hypothesis testing setting, given X = x G X, a. decision 
6{x) = corresponds to deciding in favor of a null hypothesis (i^o), whereas a decision of 
6{x) = 1, a discovery, corresponds to rejecting Hq in favor of an alternative hypothesis (Hi). 

It suffices to restrict ourselves to nonrandomized decision functions since, through the use 
of an auxiliary randomizer which is usually a standard uniform variable U that is independent 
of X, we can always convert a randomized decision function S* : (A", J-") — )■ ([0, 1], cr[0, 1]) 
into a nonrandomized decision function 6 : {X x [0, 1], J-"® o"[0, 1]) — )■ (21, cr(2l)) via 6{x, u) = 
I{u < S*{x)} with /{■} the indicator function. 

Thus, in our general formulation, the sample space X may actually represent a prod- 
uct space between a data space and [0,1]. This framework is appropriate, for instance, 
when dealing with discrete data or when using nonparametric decision functions. For more 
discussions on this matter, see [HI [TO] . 

Decision or test functions typically depend on a size parameter a G [0, 1]. For example, 
when testing the null hypothesis Hq : fi = versus the alternative hypothesis Hi : fi ^ 
based on a random observable X ~ N{fi,l), the size-a test 6 : X = ^ -^ {0,1} has 
6{x;a) = I{\x\ > $~^(1 — a/2)}, where $~^(-) is the quantile function of the standard 
normal distribution. Henceforth, in order to simplify our notation, we shall adopt a functional 
notation where 6{a) represents the statistic defined on X according to x h-)> 6{x;a). Now, 
when viewed as a process in a, we then obtain the notion of a (nonrandomized) decision 
process introduced in [2], which is a stochastic process A = {S{a) : a G [0,1]} where, 
Va G [0, 1], 6{a) is a decision function, and such that the following conditions are satisfied. 

(Dl) (5(0) = and S{1) = 1 a.e.-P. 

(D2) The sample paths a i— )■ 6{a) are, a.e.-P, {0, l}-valued step-functions which are nonde- 
creasing and right-continuous. 

Let A^ be a finite set with \A4\ = M. An A^-indexed multiple decision problem is one 
whose action space is 21^. In the context of a multiple hypotheses testing problem, for 
each m G A^, there is a pair of hypotheses Hmo and Hmi- Of interest is to simultaneously 
decide between Hmo and H^i for each m & A4. A multiple decision function (MDF) for 
such a problem is a 5 = {6m '■ rn G A^) where 6m is a decision function, so that 6 : 
{X,T) -^ (2l^^a(2t^0)- A multiple decision process (MDP) is a A = (A^ : m G Al) 
where A^ = {6m{o:) : a G [0, 1]} is a decision process. 



For each P eV, let there be subsets A^o(P) and A^i(P) of M such that 
M = Mo{P) U Mi(P) and Mo(P)nMi{P) = (H. 
We shall assume that the following condition holds. 

(D3) Under P, the subcollections {A.^ : m E A^o(P)} and {A„ : m G A^i(P)} are 
independent of each other, and the elements of {A^ : rn G A^o(P)} are independent. 

In the multiple hypotheses testing situation, Hmo is true under P if and only if m G A^o(P)- 
Observe that the elements of {A^ : m G A^i(P)} need not be independent of each other, 
under P. We shall also assume that 

(D4) With -Ep(-) denoting the expectation operator under P, then VP G V, Vm G A^o(P), Va G 
[0, 1], we have Ep {^^(a)} = a- 

The collection of all A^-indexed multiple decision processes satisfying conditions (Dl)- 
(D4) will be denoted by T). We remark that the requirement of equality in (D4) given 
by Ep {Sm{<y)} = 01 will usually be fulfilled in many situations since an auxiliary randomizer 
is incorporated in our framework, though there may still be situations when dealing with 
non-regular families of distributions where this condition may not be satisfied. The latter 
will manifest itself when the decision functions already have power equal to one but without 
yet requiring their sizes to equal one. 

Let A = (Am : m G A^) be an A^-indexed collection of measurable functions with 
Ara ■ ([0, l],a[0, 1]) — ;■ ([0, 1],(t[0, 1]). We assume that, for each m G M, the following 
conditions are satisfied: 

(Al) ArniO) = and A„(l) = 1. 

(A2) The mapping a h-> Am{a) is continuous and strictly increasing. 

(A3) Va G [0, 1], UmeMl^ - ^-(«)] > 1 - «• 

(A4) Va G [0,1], VP G V : \Mo{P)\maXmeMo{P) Ami^) < EmeM^rnia)- 

Such an A will be called a multiple decision size function. The collection of all A^-indexed 
multiple decision size functions will be denoted by (3. A particular element of 6 is the Sidak 
multiple decision size function (cf., [22]) A'^ = {A^ : m G Al) with 

A^{a) = !-(!- aY^^', ae[0,l],meM. (2.1) 

Another particular element of 6 is the Bonferroni size function A^ = {A^ : m G Al) with 

A^{a) = a/M, a G [0, 1], m G Al. (2.2) 



Before proceeding we also recall the notion of generalized P-value statistics; see 
Given a A G 2) and an A G (3, we define for m E Ai the random variable 

am = am{A, A) = inf {a G [0, 1] : (5^(A„(a)) = 1} . (2.3) 

The collection (Q;m(A, A) : m G M.) is called the vector of generalized P-value statistics 
associated with the pair (A, A). Observe that the usual P-value statistic associated with 
Sm is Pm = Ajn{am), hcucc the use of the adjective generalized ioi the ctmS. We shall assume 
without much loss of generality that these generalized P-values are a.e. \P] distinct. 

3 Main Theorems and Classes of MDFs 

We shall present in this section the two main results that will enable the construction of the 
classes of multiple decision functions controlling FWER and FDR. 

Given a A = {A^ : m G M} G D, an A = {Am : m G M] G 6, a P G P, and an 
a G [0, 1], define the stochastic processes So = {Sq^q) : a G [0, 1]}, S = {S{a) : a G [0, 1]}, 
and F = {F{a) : a G [0, 1]}, where 

So{a) = S'o(a;A,A,P)= J^ (5^(A^(«)); (3.1) 

mgXo(P) 

S{a) = Sia;A,A)=J2^rniAmia)y, (3.2) 

F{a) ^ P(a;A,A,P) = ^/{5(«)>0}, (3.3) 

b{a) 

with the convention that 0/0 = 0. These quantities have the following interpretations. Given 
an a G [0, 1], for each m G M., the decision function whose size is Am{o) is chosen from 
Am, and the MDF 5{a) = {6m[Ami<y)] : m G A^) will be employed in the decision-making. 
For this MDF (5(a), then S'o(a) is the number of false discoveries, S{a) is the number of 
discoveries, and F{a) is the proportion of false discoveries among all discoveries. Observe, 
however, that since P is unknown, both Sq and F are unobservable, whereas S is observable. 
For g G [0, 1], let us also define the random variables 

a\q) =a\q-A,A) 



inf Lg [0,1]: W [l-Am{a)Y- 



5™(A„(a)-) ^i_A. ^34) 



and 



a*(g) = a*(g;A,A) 



sup < a G [0, 1] : ^ Am{a) < qS{a; A, A) I . 

I meM ) 



(3.5) 



In essence, a\q) is a first crossing-time random variable, whereas a*{q) is a last crossing-time 
random variable. The forms of these two random variables were motivated and justified in 
Sections 6 and 7 in |T3] for a specific multiple decision size function, but the justifications in 
that paper carry over to the more general setting considered here. 

The two main results of this paper are contained in Theorem 13.11 and Theorem 13. 2[ We 
present the statements of these theorems, but defer their proofs to Section H] after some 
discussions about their implications and potential usefulness. 

Theorem 3.1 Under conditions (D1)-(D4) fori) and (A1)-(A3) for &, we have thatyP G 

P,VAe2),VAG6,VgG [0,1], 

Ep {I {So{a\q; A, A); A, A,P) > 1}} < q. 

Observe that Ep {l{Soia\q; A, A); A, A, P) > 1}} is the FWER since it is the proba- 
bility of committing at least one false discovery when the true underlying probability measure 
is P. Thus, Theorem 13. II shows that for any q G [0, 1], any multiple decision process A G !l), 
and any multiple decision size function A G 6, the MDF defined via 

6\q) = 5\q- A, A) = {5^[Am{a\q- A, A))] : m e M) , (3.6) 

strongly controls the FWER at q. 

Theorem 3.2 Under conditions (D1)-(D4) forT) and (Al)-(A4) for &, we have thaf^P G 
P,VAGD,VAG6,VgG [0,1], 

EF{F{a*{q-A,A)-A,A,P)}<q. 

Note that Ep {F{a*{q; A, A); A, A, P)} is the FDR as introduced in the seminal paper 
of [T]. The implication of Theorem 13.21 is that if, for each q G [0,1], and for any multiple 
decision process A G 2) and multiple decision size function A G 6, we define the MDF 

5*{q) = 5*iq;A,A) = i5m[Am{a*iq;A,A))]: m E M) , (3.7) 

then 6*{q) is an MDF that controls the FDR at q. 

The importance of the preceding results is that each multiple decision process A G 2) 
may have an associated multiple decision size process A = A(A) G & such that the resulting 
multiple decision functions S'^lq) or S*{q) possess some optimality property, for example, with 
respect to the missed discovery rate. To define this rate, let 

MH ^MKA.A.P) = ^""^■'7J^,y;|''""''°"' j{l>'i(P)l > 0}. (3.8) 



The quantity M{a) has the interpretation of being the proportion of missed discoveries 
relative to the number of correct alternative hypotheses. Then, for instance, the missed 
discovery rate (MDR) of the MDF in ([32D is 

Ep{M(a*(g);A,A,P)}. 

For the given A, with proper choice of A, we may be able to find an MDF that strongly 
controls the FWER or the FDR, while at the same time possessing an optimal property 
with respect to another criterion, such as having a small, possibly maximally over V, MDR. 
This idea was implemented in a more restricted setting in [HI [15] when each of the pairs of 
hypotheses contained simple null and simple alternative hypotheses. 

We note that previous works usually focussed in developing a particular MDF and then 
verifying that it controls the FWER or the FDR, such as, for example, in |I]; more com- 
prehensively, see |1]. It is our hope that by providing a class of MDFs where each member 
strongly controls the FWER, given by 

(t^ = {6\q; A, A) : A G D, A G 6} ; (3.9) 

or a class of MDFs where each member controls the FDR, given by 

r = {6*{q; A, A) : A G S), A G 6} , (3.10) 

then we will acquire the possibility of selecting from these classes MDFs possessing other 
desirable properties with respect to Type II error rates. More discussion of this issue will be 
provided in Section O 

4 Proofs of the Main Theorems 

The proofs of the two theorems are analogous to those of Theorem 6.1 and Theorem 7.1 in 
[Ti] which can be found in the supplemental article [Ibj- Note that those proofs were for 
special forms of the multiple decision process and multiple decision size function, whereas in 
the current paper we are dealing with an arbitrary element A G 25 and an arbitrary element 
A G ©. In the proofs below, we assume that A G 2) and AG© have been chosen and are 
fixed. Also, q G [0, 1] and P G "P denotes the unknown underlying probability measure. The 
dependence on (A, A, P) of some of the relevant processes and quantities below will not be 
explicitly written for brevity, unless needed for clarity. 

4.1 Of Theorem [331 

Proof: We start by defining the stochastic process Hi = {Hi{a) : a G [0, 1]} via 

H,{a)^H,{a-A,A)= J] [1 - A„(a)]i-^-(^-(")-). (4.1) 

6 



The sample paths of this process are, a.e. [P], left-continuous with right-hand limits (caglad), 
are piecewise nonincreasing, and with 

1 - a < Hi{a-) = Hi{a) < Hi{a+) 

for every a G (0, 1), where the first inequality is due to property (A3). In fact, by virtue of 
property (Al) and property (Dl), note that 

lim Hi[a) = 1 and limifi(a) = 1. 

04,0 atl 

Now, in terms of Hi, we have that 

a^{q) = inf {a G [0, 1] : Hi{a) < 1 - q} . 

Since, as pointed out above, we have 1 — a < -^1(0;), then by its definition, we must have 
a^{q) > q. This implies that 

H,{a\q))>l-q. (4.2) 

For the quantity of main interest in the theorem, we now have 

Ep[l{So{a^{q))>l}] 
= P {So{a\q)) > 1} 
= l-P{5o(«t(g)) = 0} 

= l-pj fl [6UAm{c^\q))) = 0] 

[mgA4o(P) 

The last probability cannot, however, be written as a product of probabilities since the 
Sm.{Amic(^ (q))) for m G A^o(P) need not be independent owing to the dependence on a^{q) 
which is determined by all the (A^, m & Ai). On the other hand, we do have the set equality 



n [6^{Am{a\q))) = 0] = \ a^{q) < min a^ > 



(4.3) 



meXo(P) 

where the a^s are the generalized p- value statistics defined in fl2.3|) . 
Next, define the stochastic process H2 = {H2{a) : a G [0, 1]} via 

H2(a) = H2(a- A, A,P) 



1 ■^^^ -^ -^5 ■ 



n [1 - AUa)] n [1 - A«(«)]'"''"^^'"^"^"^ 

'meA^o(P) / \meA^i(P) 
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Analogously to the Hi process, this has caglad sample paths. Let us then define the quantity 

a*{q) = a*{q; A, A, P) = inf {a E [0, 1] : i^ala) < 1 - g} • 

Note that this is not a random variable since this depends on the unknown probability 
measure P, in contrast to a^iq)- Furthermore, also note that 

H2{a*{q))>l-q. (4.4) 

From their definitions, Hi{a) > H2{a), so that Hi{a) < 1 — q implies H2{a) < 1 — q. 
Consequently, 

a\q) > a*{q). (4.5) 

Now, the importance of the quantity a*(g') arises because of the crucial set equality 

aUq) < min am \ = { ot'^iq) < min «„ > . (4.6) 

mGA^o(P) J I meA4o(P) J 

To see this equality, first observed that the inclusion C immediately follows from (14. 5p . To 
prove the reverse inclusion, since 

{a*(g) < min a^} 

meA^o(P) 

imphes that, for some ao < min^eXoCP) ^^^ ^^ have H2{aQ) < 1 — g. But for such an aoi we 
will have (5m(Am(ao)~) = for all m G A^o(P)) so that 

a^e {a e [0, 1] : Hi{a) < 1 - g}. 

Consequently, 

a\q) = inf{« G [0, 1] : Hi{a) < 1 — g} < ao < niin otm- 

mgXo(P) 

The reverse inclusion D thus follows, completing the proof of (14.61) . 

By (14.31) . (14.61) . and the iterated expectation rule, it now follows that 

P \ n [5m{Am{a\q))) = 0] 
= P{a\q) < min am} 

m€Mo(P) 

= P < a'^iq) < min a^ 

I m&Mo(F) 



Er 



P < a*(q) < min am I o*{q) 

' mgA4o(P) ' 



Since (y*{q) is measurable with respect to the sub-cr-field <j{Sm '■ rn G A^i(P)), whereas 
rammeMoiF) Q^m is measurable with respect to the sub-cr-field a{6m '■ rn G A^o(P))) then 
by condition (D3), a*(g') and minmeAio{P) Q^m ^^6 independent. Furthermore, by condition 
(D3), we obtain 

P \ mm am > w\ = P I Pi [5m{Am{w)) = 0] 

= n P{Sm{AmH) = 0} 

meMo(P) 

= n [i-^-H]' 

mGA4o(P) 

with the last equality a consequence of condition (D4). Therefore, 

P I fl [SmiAmic^Hq))) = 0] 

\^meMo(F) 

= Epi n [l-^n.(«*(?))] 

I \meMo(P) 



n [1-A„(«#(g))] 

= Ep{H2{a*{q)} 
> Ep{l-q) 

= 1-g 



l~Sm{A,n(a#{q))-) 



with the last inequality following from f l4.4p . Thus, finally, we have 

Ep[l{So{a^{q))>l}] = 1-P| n [5m{Am{c^Kq))) = 0] 

\^meMo{P) 

< i-(i-g) 
= q- 



This completes the proof of Theorem 13. II || 

We remark that condition (D4) can be weakened to just having 

Vm G Mo{P),\/a e [0, 1] : ^p{(5^(a)} < a (4.7) 

to still get the desired strong FWER control. This is so since in the portion of the proof 
where we have 

m&Mo{P) meMo{P) 

we simply replace the second = sign by > and then the proof of the theorem goes through. 

4.2 Of Theorem [3:21 

Proof: This proof closely mimics that of Theorem 7.1 in [13] as presented in [Kj. As an 
aside, we mention that the seed of the idea of providing a class of FDR-controlling multiple 
decision functions was planted upon the realization that the proof of this Theorem 7.1 is 
independent of the choice of the multiple decision size function. 

The case with q = is trivial since then a*(0) = 0, so that F{a*{0)) = 0. Thus we 
restrict to g G (0, 1]. By the defining property of a*{q) given in f l3.5p . we have that 

S{a*{q)) > -^Ma*{q)) (4.8) 

where A,{a) = ^^g^ Am{(y)- Consequently, from (13. Sp . 

F{a g < q-—--—I{S{a [q) > 0} < g ■ 4.9) 

A,{a*{q}) A,{a*{q}) 

For a G [0, 1], define the sub-a- field 

T^ = T^iA, A) = a {6UAmm- Pe[a,l],meM}. (4.10) 

Observe that 5^ = (J-'a : a G [0, 1]) is a decreasing collection of sub-a-fields of J-". By its 
definition a*{q) is an 5^-stopping time. 

Let us define the process Tq = (To(q;) : a G [0, 1]) according to 

To{a) = To{a; A, A,P) = 2_^ 



AmM 
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Fix < a < f3 < 1. Then, since Sm G {0, 1}, we have 
Ep{To{a)\J^f,} 



A^(a) 



P{5UAUP)) = l\J'f3}x 



meMo{P) 

E 

meAlo(P) '" 

Ep {5„(A„(a))|(5„(A„(/3)) = 1} 

E ^™(^™(/3))ir^44^, a.e. [P] 

meAlo(P) 

To(/3). 



A„(a)A„(/3) 



The second equahty follows from (D3), whereas the second-to-last equality follows since 

Ep {5miArr,ia))\5raiA„0)) = 1} 

P{6UAm{(^)) = l,SUA„m) = '^} 

p{suAmm = 1} 

P{6UAm{(^)) = 1} 
P{6m{AUP)) = 1} 

_ Amja) 
AU/3) 

because of condition (A2) for the Am{-)s and conditions (D2) and (D4) for the (5m(-)s. The 
above results show that, under P, 

{(To(«),J-«): ae [0,1]} 

forms a reverse martingale process. Further, observe that To(l) = |A1o(P)| a-e. [P] due to 
conditions (Dl) and (Al). Thus, 

EpiToil)) = \MoiP)\. 
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From the inequality in fl4.9p . we obtain 

'So{a*{q)) 



Ep[F{a*{q))] < qE^ 



_A.{a*{q))^ 



q J2 ^P 

meMoiP) 

q Yl ^P 

meMo(P) 



< q 

< q- 

= q 

= q, 



sup 



max 



ae[0,l]m€Mo{F) A,{a) 

^ r^p[ro(l)] 



Sm{(^*{q)) 
A,{a*{q))_ 

5m{a*iq)) Arr,{a*{q)) 
A^{a*{q)) A,{a*{q)) 

"^-^"^ i?P [To(a*(g))] 



|A^o(P) 
|.Mo(P) 
l-Mo(P) 



where the last inequality is obtained using condition (A4) and by invoking the Optional 
Sampling Theorem for (reverse) martingales (cf., [3]), and the second-to-last equality because 

ofEp[To(l)] = |A^o(P)|. 

Note that, in particular, since the Sidak multiple decision size function A"^ always satisfies 
condition (A4) for all P G P, then VA G 2), VP G V, we have the property 



Ep{F{a*{q-A,A'y,A,A')}<q. 



(4.11) 



Let us denote hjVo = {P eV : Mo(P) = M}. Observe that for P G Vq, condition (A4) 
will not be satisfied unless the multiple decision size function A has identical components, 
in essence, a Sidak multiple decision size function form. We still therefore need to establish 
that for an arbitrary A G © and a P G Po) 

Ep{F{a*{q-A,A)-A,A)}<q. 

For such a P G "Po, we have F{a; A, A) = I{S{a; A, A) > 0}, so that 

Ep[F(a*(g;A,A);A,A)] 
= P{5(a*(g;A,A);A,A)>0} 
= PK(g;A,A)>0}. 

We have, for any A G 2} and any A G 6, that 



{a-(g;A,A)>0} 



u 

aG(0,l] 



S{a;A,A) ^ 1 
A,{a) ~ q 



(4.12) 
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In Lemma D.l of [T3] it was established, using an inequality of [TT], that for Wm{f]m),''T^ ^ 
A^, independent Bernoulli (77^) random variables with 77^ € [0, 1] and satisfying nmeA^(^ ~ 
rjm) = 1 — a, for each t > 1, 

p J l^meM '^mJVm) y, A < p J ^meM '^rn[Vrn) y, A ('4 13') 

where 77^ = 1 — (1 — a)^/^^, m e M. 

Noting that, under P G Po, '5m(^m(a))s are independent Bernoulli (Am (a)), then by 
using the inequality in f l4.13p and condition (A3), it follows that for q G (0, 1], 

\ A.{a) qj \ AS+{a) qj' 

where the Sidak sizes A'^"'" = {A^,m G A^) in ( I4.14p have components 

A^ = 1 - {1 - a+y/^^ ,m e M, 

with a"*" satisfying HmGA^l-'^ ~ Am(")] = 1 — "^- Observe that by (A3), we have necessarily 
that a"*" < a. Combining the results in (14.121) and (14.141) . we obtain 

P{a*{q] A, A) > 0} < P{a*{q; A, A^+) > 0}. 

But since we have already established that, for P G Po? we have 

PK(g;A,A^+)>0}<g, 

then it follows that P{a*(g; A, A) > 0} < q. This implies finally that 

Ep{F{a*{q;A,Ay,A,A)}<q 

for any P G Vq, thereby completing the proof of Theorem 13.21 || 

In contrast to Theorem 13.11 where we were able to have the weaker version of condition 
(D4) given in (14. 7p . we could not do this for Theorem 13.21 The reason is that we could 
not conclude under this weaker condition that the process {(To(a), J-'q,) : a G [0,1]} is a 
reverse supermartingale, which would have allowed us to get the desired result. It may be 
possible that under certain situations we do have this supermartingale property, but the 
weaker condition (14. 7p appears not sufficient for this property to hold in general. 
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5 Representations of MDFs in Terms of the General- 
ized P- Values 

This section expresses the MDFs 5^(g) in fl3.6p and S*{q) in (13 .yp in terms of the generahzed 
p- value statistics defined in fl2.3p . Define the anti-rank statistics vector via 

((1), (2), . . . , (M)) : {X, ^) ^ {m, a{m)) (5.1) 

where 9Jt is the space of all possible permutations of Ai, and such that 

"(1) < "(2) < . . . < a(Af)- 

Let us first consider the random variable a'^lq) in (13.41) . We see from its definition and 
those of the generalized p-value statistics that, for some J E M = {0} U Ai, we have 

a^(g) G [a(j),a(j+i)) 

if and only if 

Vj G {1, 2, . . . , J} : UmeMi^ " ^M ("O')] '"'""> '^"">^"'^'^^ > 1 - ?! 

From the definition of the generalized p-value statistics we further have 

6(m)[A(m)ia(j))-] = I{m < j - 1}. 
Consequently, by defining the TW-valued random variable 

Jt(g) = max J A; G 7W : f[ [1 - A^^n) (ao))] > 1 - q,J = 1,2, . . . ,k[ , (5.2) 

I rn=j ) 

we have the result that 

a\q) G [a(jt(g)),a(jt(g)+i)) . 

As a consequence we obtain the representation of 5^(q') in (13. 6 p in terms of the a^s given by 
SHq) = {5m{AUa\q))),me M) = (5™(A„(a(jt(g)))),m G M) , (5.3) 

where we used the fact that, for each m E Ai, 6jn is constant in the interval 

[^m(a{Jt(q))), ^m(a{Jt(g)+l))- 
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Next let us consider the random variable a*{q) in fl3.5p . We may re-express its defining 
equation via 

a*{q) = sup < a G [0, 1] : ^ v4(„)(a) < q ^ 6(m)[A(m){a)] 

K meM meM 

But, since Emex ^m[^m("0))] =J^ ^^^n 

a*{q) E [a(j),a(j+i)) 
if and only if 

Vj G { J + 1, J + 2, . . . , M} : E^gA, ^m(«0)) > gj. 



Defining the A^-valued random variable 

l(m)(a(fe)) <qk} , (5.4) 

we then have that 



J*{q) = max <keM: ^ ^( 



a*{q) G [a(J-(g)),a(J-(5)+i)) • 

As a consequence, an equivalent representation of the MDF 6*{q) in fl3.7p in terms of the 
OmS is provided by 

6*{q) = {6m{A^{a*{q))),m e M) = (5^ (A^(a(j* (,)))), m G TW) . (5.5) 

The representations in (15.31) for (5^(g) and (15.51) for 6*{q) provide alternative computational 
approaches since, instead of computing a^(g) and a*{q), we may simply compute the gen- 
eralized p- values, then J^(q') and J*{q), and then finally the realizations of the decision 
functions. 

For a simple application, let us see what becomes of the MDFs S'^i^q) and S*{q) if we use the 
Sidak multiple decision size function A'^ given in (12. ip . We use the alternate representations 
just obtained above. By simple manipulations, we immediately obtain that 

Jt(g) = max {fc G A^ : a(j) < 1 - (1 - g)*^/(*^-J+i), j = 1, 2, . . . , A;} ; 
J*{q) = max {k e M : M[l - (1 - a(fc))]^/*^ < qk} . 

But, for these Sidak size functions, the (ordinary) p-value statistics are given by 

Pm = Alia^) = 1 - (1 - a^)i/*^m G M. 
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Re-expressing the J^{q) and J*{q) in terms of these p- values, we easily obtain by simple 
manipulations that 

Jt(g) = max {A; G M : Pq) < 1 - (1 - g)V{M-i+i)^^- = i, 2, . . . , A;} ; (5.6) 

J*{q) = max {k E M : P^k) < qk/M] . (5.7) 

Observe that J^(g) in (15. 6 p leads to the step-down sequential Sidak FWER-controlling pro- 
cedure, see |1]; whereas, J*{q) in (15.71) is the usual form of the step-up Benjamini-Hochberg 
FDR-controlling procedure in [1]. Thus, through the Sidak sizes, we are able to obtain from 
our formulation two popular MDFs for FWER and FDR control as special cases of the MDFs 
5\q) and(5*(g)! 

6 Towards the Development of Optimal MDFs 

Finally, in this subsection, we indicate, without going into much detail, the potential utility 
of the classes of MDFs arising from Theorems 13.11 and 13.21 in the context of obtaining MDFs 
with some optimality properties, especially in non-exchangeable multiple hypotheses testing 
settings, which are those where the power characteristics of the M test functions are not 
identical. 

Let us fix a multiple decision process A G 25 and fix a probability measure Pi G V. 
Define the mappings Hm '■ [0, 1] — ;■ [0, 1] for m G A1 according to 

7r^(a;Pi) = EpJ5^(a)], aG[0,l]. (6.1) 

When viewed as a function of Pi, vrm(a; ■) is the power function of 5m when it is allocated 
a size of a. Of interest to us, though, is to view it as a function of a for the fixed Pi. 
In this case, 7rm(-;Pi) is the receiver operating characteristic curve (ROC) of the mth test 
function. Assume that for each m G A^, the mapping a i— )■ vrm(a; Pi) is strictly increasing 
with 7rm(l; Pi) = 1 and twice-differentiable. 

Suppose it is desired to strongly control the overall FWER or FDR at some pre-specified 
level q G [0, 1], but at the same time maximize the total (or average) power at P = Pi. Our 
idea, partly implemented in P3] , is to first obtain the optimal multiple decision size function 
for weak FWER control associated with A, denoted by A* = {A*^{a)^m G M) G &. This 
is the multiple decision size function A satisfying the condition 

V« G [0, 1] : n [^ " ^"(")] = 1 - «, 

meM 

and such that the total power at P = Pi, given by ^m£_A4 7'"m(^m(«); Pi), is maximized. 
Under regularity conditions on the ROC functions, the optimal A* function could be ob- 
tained using Lagrangian optimization, for instance, see Theorem 4.3 in [13] which is an 
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implementation when the individual test functions coincide with the Neyman-Pearson most 
powerful tests. 

Now, having determined the optimal multiple decision size function A* associated with 
A, which is at this point optimal only in the sense of weak FWER control, we can then 
apply Theorem 13.11 to obtain the MDF (5^(g; A, A*) which will strongly control the FWER 
at g; or apply Theorem 13.21 to obtain the MDF 6*{q] A, A*) which will control the FDR at 

By virtue of the choice of the size process A*, which is tied- in to the multiple decision 
process A and the target probability measure Pi, we expect that the MDFs (5^(g; A, A*) and 
S*{q; A, A*) will perform better with respect to overall power at Pi relative to, for example, 
the sequential Sidak MDF or the BH MDF, which we saw from the preceding section are 
MDFs arising from the Sidak multiple decision size function, a size function that may not 
be optimal for the chosen multiple decision size process A. For instance, results of a modest 
simulation study in [T3] demonstrated the improvement over the BH procedure of the MDF 
S* in a specific setting. Further improvements in power performances could be achieved by 
proper choice of the multiple decision process A, such as, for example, choosing it to have 
components that are uniformly most powerful (UMP) or uniformly most powerful unbiased 
(UMPU) test functions. These issues, however, will be deferred for future work, but we 
expect that the classes of MDFs presented here will play a central role in dealing with these 
more complex multiple hypotheses testing problems. 

We close by pointing out that other approaches have also been proposed for obtaining 
MDFs possessing certain optimality properties. Relevant papers pertaining to optimality are 
[2S1 UM El El E31 EH EHl [121 [IS] • Procedures with a Bayes or an empirical Bayes flavor can 
be found in [131 HH [3, [9]. In addition, it is also of interest to extend our results to settings 
where the components of {Sm '■ rn e A^o(P)} are dependent as in [HI [2]; see also the review 
article [18]. Another possible extension is to consider generalized FWER and FDR as in 
[T7] . However, we defer consideration of such extensions for future work. 
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